Description

• Fixes root cause for flakiness in test_crawler_statistics_persistence
• Previously, BasicCrawler was manually emitting a persist state event when exiting. Now this should be done by the EventManager.
• Statistics used to be double-persisted previously, and due to a race condition, the old state could be persisted sometimes. Now the statistics are persisted by themselves when exiting their own context, and not by the Crawler-emitted event
• StatisticsState.crawler_runtime changed to a computed field in a backwards compatible way. This allows pushing the runtime calculation to the state and ensures consistency between attribute access and persistence. (This prevents a theoretical race condition when automatic persistence triggers almost at the same time as the crawler finishes, and again causes the scenario described above).

Issues

• Closes: Test test_crawler_statistics_persistence is flaky #1560

Testing

• Stress testing in CI
• Added unit test

Checklist

• CI passed